1. Field of the Invention
This invention relates to sound-picture synchronous compression and synchronous reproduction systems in which sound information and picture information are subjected to synchronous compression and synchronous reproduction in a real-time manner. Specifically, the sound-picture synchronous compression and synchronous reproduction system (hereinafter, simply referred to as a sound-picture synchronous compression-reproduction system) is actualized using a sound-picture synchronous compression device and a sound-picture synchronous reproduction device. This invention is based on patent application No. Hei 9-27140 filed in Japan, the content of which is incorporated herein by reference.
2. Prior Art
The sound-picture synchronous compression-reproduction system is actualized by a pair of sound-picture synchronous compression device and sound-picture synchronous reproduction which use CPUs (an abbreviation for "Central Processing Unit") as well as DSPs (an abbreviation for "Digital Signal Processor"). In the sound-picture synchronous compression device, sound information and picture information are subjected to synchronous compression in real time. So, the compressed sound information and compressed picture information are transmitted via transmission lines to the sound-picture synchronous reproduction device, wherein they are subjected to synchronous reproduction in real time.
Normally, sounds and pictures are transmitted in a digitized manner via telephone lines and are then recorded on recording media by using compressive encoding because an amount of data thereof is enormous. The compression technique for the pictures frequently uses an encoding method designed on the basis of the DCT (an abbreviation for "Discrete Cosine Transform"), which uses a property that spatial frequencies of the pictures tend to be concentrated into low-frequency ranges. This encoding method is employed by the international standard, namely, MPEG (an abbreviation for "Motion Picture Experts Group") Video, recommendation H. 263.
In addition, the compression technique for the sounds frequently uses a so-called perception encoding method which omits inaudible information, which cannot be heard by human ears, by using auditory psychology. This encoding method is employed by the international standard, namely, MPEG Audio, recommendation G. 723.
FIG. 10A to FIG. 10F show hierarchy for encoding formats based on the standard of MPEG Video. Codes of the MPEG Video are assigned to levels of a hierarchical structure as shown in FIG. 10A to FIG. 10F. A top level of hierarchy corresponds to a video sequence which is configured by multiple GOPs (an abbreviation for "Group Of Picture"). Each GOP is configured by multiple pictures.
There are provided three kinds of pictures, namely, I picture, P picture and B picture. Herein, the I picture corresponds to intra-frame codes; the P picture corresponds to interframe codes with respect to a forward direction only; and the B picture correspond to interframe codes with respect to both of forward and backward directions. The picture consists of multiple slices corresponding to regions which are determined by partitioning an overall area of the picture. Each slice is configured by multiple macro blocks which are arranged from the right to the left or from the top to the bottom.
A block of 16.times.16 dots (or pixels) is divided into blocks of 8.times.8 dots, which are used for representation of luminance components (i.e., Y1, Y2, Y3 and Y4). In addition, blocks of 8.times.8 dots are used for representation of chrominance components (i.e., Cb, Cr) in connection with the region of the luminance components. Thus, the macro block is constructed by six blocks of 8.times.8 dots in total. Therefore, the block of 8.times.8 dots is used as a minimum unit for encoding.
FIG. 11 shows an example of the hierarchy for encoding formats based on the standard of MPEG Audio. Codes of the MPEG Audio are constructed using "AAU" (an abbreviation for "Audio Access Unit"), which is a minimum unit for encoding.
Now, a description will be given with respect to the conventional technique for compression of sound codes and picture codes. FIG. 12 is a block diagram showing an example of a sound-picture synchronous compression device which performs synchronous compression and transmission on sounds and pictures. In FIG. 12, a CPU 2 executes programs of a device control block 4 so as to control the device as a whole. A user operates a keyboard 3 to input commands and instructions to the device. States of execution of the programs are visually displayed on a screen of a display 1.
Picture data are taken by a camera 6 and are input to the device by means of a picture input block 5. Sound data are created by a microphone 8 and are input to the device by means of a sound input block 7. The sound data are subjected to compression by a sound compression block 10. The picture data are subjected to compression by a picture compression block 9. The compressed sound code corresponding to the compressed sound data as well as the compressed picture code corresponding to the compressed picture data are mixed together using a single code by a code nixing block 11. Then, mixed codes are transmitted onto a transmission line by a transmitter block 12 as well as a modem 13.
FIG. 13 is a block diagram showing an example of a sound-picture synchronous reproduction device which performs receiving and synchronous reproduction on sound-picture codes transmitted thereto. In FIG. 13, a CPU 22 executes programs of a device control block 24 to control the device as a whole. A user operates a keyboard 23 to input commands and instructions to the device. States of execution of the programs are visually displayed on a screen of a display 21.
A receiver block 32 receives mixed codes transmitted thereto from a modem 33 as sound codes and picture codes to be reproduced. Then, a code separation block 31 separates the mixed codes into sound codes and picture codes respectively. The sound codes are subjected to expansion (or reproduction) by a sound reproduction block 30. The picture codes are subjected to reproduction by a picture reproduction block 29. Then, contents of picture data corresponding to the reproduced picture codes are visually displayed on a screen of a monitor 26 by a picture output block 25. A sound output block 27 provides a speaker 28 with sound data corresponding to the reproduced sound codes. So, the speaker 28 produces the corresponding sounds.
The aforementioned example of the system for sound-picture compression and reproduction performs communications of codes. Herein, data processing should be performed in a real-time manner. In general, a number of operations (or calculations) are required in processing for the compression and reproduction. To enable real-time processing, the conventional system which uses the software for compression and reproduction of sounds and pictures reduces a number of frames representing pictures which are subjected to processing. Herein, parallel operation instructions of the CPU and DSP are used to perform multiple operations at once, so it is possible to accomplish high-speed performance in the above processing.
FIG. 14 shows an example of the parallel operation instructions. Herein, a 64-bit register R0 stores 4-word values a3, a2, a1 and a0, while a register R1 stores 4-word values b3, b2, b1 and b0. Those 4-word values are added together, so addition results are stored in the register R0. According to the parallel operation instructions described above, addition is performed with respect to four words at once, so it is possible to perform high-speed processing.
Functions of real-time OS (an abbreviation for "Operating System") can be used for the compression and reproduction of the sounds and pictures. Herein, if the sound processing and picture processing are switched over periodically by a certain period of time, it is possible to perform real-time processing with ease. At a time to switch over the processing, contents of the registers which are presently used should be retained. For this reason, the system performs processing in parallel, and implements the parallel operation instructions requiring "large-bits" registers each having a large number of bits. In that case, the switching of the large-bits registers frequently occurs, so it is impossible to perform high-speed processing. To avoid such a problem, it is necessary to add an additional process which initiates the switching of the processing between the sound compression and picture compression at the appropriate timing so that one of them is selectively performed.
It is possible to list some papers which disclose conventional examples of the system which relates to synchronous processing of the sound and picture.
For example, the paper of Japanese Patent Laid-Open Publication No. 7-64730 (denoted by "paper 1") discloses a conventional example of the system which processes sounds and pictures in a synchronized manner. The content of the paper 1 is summarized as follows:
A buffer memory is normally placed in a data accumulated state so that it is normally filled with a certain amount of data or more. A reproduction portion of one side always issues a data transfer request, so that data are transferred to reproduction portions of both sides by a unit of a data block.
The paper of Japanese Patent Laid-Open Publication No. 7-75059 (denoted by "paper 2") discloses a conventional example of the system, the content of which is summarized as follows:
The system checks a difference between a reproduction time of the sound and a reproduction time of the picture. If the sound is delayed from the picture in reproduction, a same screen image is continuously displayed. If the picture is delayed from the sound in reproduction, interpolation is performed with respect to the sound.
The paper of Japanese Patent Laid-Open Publication No. 7-110756 (denoted by "paper 3") discloses a conventional example of the system, the content of which is summarized as follows:
The system checks a time required for reproduction of one frame. Then, the system sets and processes a picture which should be displayed for a next frame. So, the system reproduces the sound accompanied with the picture.
The paper of Japanese Patent Laid-Open Publication No. 7-184143 discloses a conventional example of the system, the content of which is summarized as follows:
The processing of the system is controlled to be suited to the "long" processing whose processing period is long. Herein, after performing the long processing whose processing time is long, the system performs the "short" processing whose processing time is short multiple times by a constant period of time.
The aforementioned examples of the sound-picture synchronous compression-reproduction system may have a large interval of time for switching over the sound processing and picture processing. In such a case, the system cannot catch up with the timing of execution of the sound processing. In the case where the sound and picture operate in parallel, an overhead for switching over the sound processing and picture processing becomes large. For this reason, the conventional system suffers from a problem that it cannot catch up with the timing of execution of the sound processing, so the reproduction breaks off. In the case where adjustment is performed on the switching timing between the sound processing and picture processing in response to the performance of the CPU and DSP as well as the transmission speed of the communication lines, it is necessary to modify the system with respect to the sound processing and picture processing. In that case, there is a problem that adjustments of the programs take much time.